Today's computer display systems are capable of generating very high quality images. By contrast, in the early 1980s, the color displays commonly available with computers offered only 320 by 200 pixel resolution, with a maximum of four colors. During the 1980s, however, increasingly better computer graphics systems became available that offered higher resolution and a much larger number of colors. However, the improved systems did present some disadvantages. Not surprisingly, high quality graphics adapters and monitors were relatively expensive. Further, early high resolution monitors were capable of working only with one type of graphics adapter.
The advent of multi-scanning monitors relieved part of this concern. Multi-scanning monitors are capable of displaying images at a variety of resolutions and include control systems that can detect the resolution generated by a graphic display source, adapt the scale of the image data received to the native resolution of the monitor, and generate a refresh clock to display an appropriate image. Thus, for example, if a multi-scanning monitor capable of generating an 800 by 600 pixel image receives image data for a 640 by 400 pixel resolution image, the multi-scanning monitor rescales the image to generate a screen-filling 800 by 600 pixel image from the 640 by 400 pixel resolution. In addition to correcting images for differences in scale, improved monitors also correct gain and gamma levels to adjust for the monitor's operational parameters.
However, the improved display capabilities of display monitors adds complexity to the display circuitry. Circuitry required to adjust the displayed image adds cost to the display. Moreover, the additional processing involved in, for example, rescaling the image, delays the image generation process. Although in many applications, a slight image generation delay does not pose a problem, in computer games, latency will likely pose a concern, because it detracts from the enjoyment and naturalness of the machine-human interaction. The latency of a system in responding to user input on the display can cause the human interaction in a virtual environment or electronic game to appear disconnected, if the delay between a user input and its effect on the virtual environment is sufficiently long to be noticeable.
Newer interactive systems that are being developed may be particularly susceptible to problems with latency, since the input is primarily through a display surface in such systems. For example, the MIT Media Lab, as reported by Brygg Ullmer and Hiroshi Ishii in “The metaDESK: Models and Prototypes for Tangible User Interfaces,” Proceedings of UIST 10/1997:14-17,” has developed a form of “keyboardless” human-machine interface. The metaDESK includes a generally planar graphical surface that not only displays computing system text and graphic output, but also receives user input by responding to an object placed against the graphical surface. The combined object responsive and display capability of the graphical surface of the metaDESK is facilitated using infrared (IR) lamps, an IR camera, a video camera, a video projector, and mirrors disposed beneath the surface of the metaDESK. The mirrors reflect the graphical image projected by the projector onto the underside of the graphical display surface to provide images that are visible to a user from above the graphical display surface. The IR camera can detect IR reflections from the undersurface of an object placed on the graphical surface. Others have been developing similar interfaces that react to a user engaging an interactive display. For example, papers published by Jun Rekimoto of the Sony Computer Science Laboratory, Inc., and associates describe a “HoloWall” and a “HoloTable” that display images on a surface and use IR light to detect objects positioned adjacent to the surface.
In these situations where the system responds to a user's interaction with graphics presented on a display, latency results in the response of the system lagging behind a user's actions in providing an input. Clearly, the more rapidly images are generated and regenerated in response to a user's interaction, the more satisfying the user's experience will be. On the other hand, if a user moves a physical object across the interactive display, and there is a noticeable delay in the system's response to that movement, the system will be less effective and less enjoyable to use.
In such systems, detecting the user's interaction with the display may present a relatively complex computational task. Moreover, whatever response the computing system needs to substantively respond to the user's interaction may represent another computational burden. These computational tasks may take some time to perform. Thus, to reduce overall system latency, once the user's actions have been detected and the response generated, it is important that the display system not further delay the presentation of that response. Thus, it is highly desirable to be able to streamline operation of an image display system to reduce latency, particularly in interactive display systems in which it is desirable for the user to feel that the interaction is directly with a virtual environment being presented on the graphic display surface.